Log-Linear Models

This is yet another introduction to log-linear (“maximum entropy”) models for NLP practitioners, in the spirit of Berger (1996) and Ratnaparkhi (1997b). The derivations here are similar to Berger’s, but more details are filled in and some errors are corrected. I do not address iterative scaling (Darroch and Ratcliff, 1972), but rather give derivations of the gradient and Hessian of the dual objective function (conditional likelihood). Note: This is a draft; please contact the author if you have comments, and do not cite or circulate this document. 1 Log-linear Models Log-linear models1 have become a widely-used tool in NLP classification tasks (Berger et al., 1996; Ratnaparkhi, 1998). Log-linear models assign joint probabilities to observation/label pairs (x, y) ∈ X× Y as follows: Pr ~ θ (x, y) = exp ( ~ θ · ~ f(x, y) ) ∑ x′,y′ exp ( ~ θ · ~ f(x′, y′) ) (1) where ~ θ is a R-valued vector of feature weights and ~ f is a function that maps pairs (x, y) to a nonnegative R-valued feature vector. These features can take on any form; in particular, unlike directed, generative models (like HMMs and PCFGs), the features may overlap, predicting parts of the data more than once.2 Each feature has an associated θi, which is called its weight. Maximum likelihood parameter estimation (training) for such a model, with a set of labeled examples, amounts to solving the following optimization problem. Let {(x1, y∗ 1), (x2, y∗ 2), ..., (xm, y∗ m)} ∗This document is a revised version of portions of the author’s 2004 thesis research proposal, “Discovering grammatical structure in unannotated text: implicit negative evidence and dynamic feature selection.” Such models have many names, including maximum-entropy models, exponential models, and Gibbs models; Markov random fields are structured log-linear models, conditional random fields (Lafferty et al., 2001) are Markov random fields with a specific training criterion. The ability to handle arbitrary, overlapping features is an important advantage that log-linear models have over directed generative models (like HMMs and PCFGs). Importantly, for the latter type of models, maximum likelihood estimation is not straightforward when complicated feature dependencies are introduced that cannot be described as a series of generative steps (Abney, 1997). This comparison will be more fully explored in my thesis.